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ABSTRACT 

This paper reports on the Colorado Commission on 
Higher Education (CCHE) and their progress in implementing their 
goals as set by the state legislature which concern efforts to assess 
students knowledge capacity, and skills. In particular the report 
describes CCHE efforts to monitor institutional implementation 
efforts, summarizes institutional progress in assessment, and lists 
issues that emerged from institutional reports. A first section, a 
review of institutional reports, describes data to support the 
process used by CCHE to monitor institutional compliance with the 
accountability statute and discusses the criteria that each 
institution report on with a chart to provide detailed information. 
Also included are a description of the written review of each 
institution's report with a sample of the CCHE questions and 
institutional answers. The second section considers issues derived 
from the institutional accountability reports including supporting 
data, recommendations, and designation of a responsible agency to 
carry out the recommendations. The specific issues covered in this 
section are effectiveness of basic skills, assessment of teacher 
education programs, and minority retention and graduation^ Appended 
information includes a chart on how students are informed of 
institutional expectations, a chart on information dissemination, and 
estimated costs of accountability by institution. (JB) 
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EXECUTIVE SUMMARY 



This report responds to the General Assembly directive to the Colorado 
Commission on Higher Education (CCHE) to develop policies, monitor 
implementation and report annually about public higher education institutional 
efforts to assess their students "knowledge, capacity, and skills/ The CCHE 
developed policies and criteria to respond to the directives in the statute* The 
report 1) describes CCHE efforts to monitor institutional implementation efforts, 
2) summarizes institutional progress in assessment and 3) lists three issues that 
emerged from the institutional reports. 

Colorado public higher education institutions have documented an increase in 
students' knowledge, skills and capacities as the result of taking classes, finishing 
general education curricula, and receiving degrees in disciplines or vocational 
programs. Institutions provided data for the following report based on their efforts 
to measure their students' abilities. All institutions are implementing an 
accountability plan according to statute and policy. 

Institutional reports have some common themes. Many institutions found that 
advising is an area that students and alumni find deficient. To remedy this 
problem, institutions are providing more information and training to faculty. Most 
institutions find that their student retention rates are lower than expected and need 
to be increased. Academic alert systems that identify students with academic 
dif^culties early in a semester, and freshman seminars to introduce students to 
college life and available services are strategies to increase retention. 

Three frequent themes that have statewide implications emerge from the 
institutional reports: basic skills effectiveness, assessment of teacher education 
programs, and minority retention/graduation. Since HB 1187, the Commission has 
targeted each of these areas for further study because of their importance to the 
state as a whole. The report recommends that institutions or governing boards 
study these areas to clarify the results of assessment measures. The CCHE will 
continue to monitor these areas, including institutional and governing board 
response. Modifying CCHE policies could be one response to needs identified 
by further study of these statewide issues. 



UCTION 



In HB 1187, the Colorado General Assembly directed the Colorado Commission 
on Higher Education to develop policies and monitor the implementation of 
accountability plans at public higher education institutions. In addition, the 
Commission must report the status of institutions' implementation efforts to the 
Gene^ Assembly annually- This report fulfills that mandate. 

The report has two sections: 1) a description of and data to support the process 
used by CCHE to monitor institutiond compliance with the Accountability statute, 
and 2) statewide policy issues derived from the institutional accountability reports. 
The first secdon discusses the criteria that each institution must report to CCHE 
with a chart to provide detailed information. A description of the CCHE written 
review of each im:t7tution's report with a sample of CCHE questions and 
institutional answers completes the first section. 

The second section lists three statewide policy issues derived firom the institutional 
reports. Each issue is described, supporting data supplied, recommendations 
discussed and a responsible agency (designated to carry out those recommendations. 

In addition to this report, each legislator was sent a copy of a two-page summary 
developed by each institution in the legislator's district about its accountability 
efforts. The two-page summaries provide specific information about each 
institution's plan and results. Additional copies of the summaries and each 
institution's complete report are available from CCHE. 

INSTITUTIONAL REPORT REVIEW 

Statutory Directives and Criteria 

CCHE developed a set of criteria based on the four directives in the statute to 
review each accountability report. The four statutory directives and their related 
criteria are presented on the next page. 
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Statutory Directives 


Criteria jj 


CRS 23-13-101 (a) "That institutions of 
higher education be held accountable for 
demonstrable improvements in student 
knowledge, capacities, and skills 
between entrance and graduation;" 


Does the institutional report: 1 

1) document changes made as a 
result of assessment? 

2) provide evidence of continuing 
examination of existing goals? 

3) contain current data on the 
assessment of general education, 
discipline-specific education, 
retention and comDletion. 
alumni/student satisfaction, and 
after-graduation performance? 


CRS 23-13-101 (b) "That these 

QCinonbirauiC iiiipruvciiftwiii5 uc puuiiciy 
announced and available;** 


Does the institutional report record how 

tVi^ inctitiitiori niihlipiv di^^minflt^H thfi 

Student assessment results? 


CRS 23-13-101 (c) **That institutions 
express clearly to students their 
expectations of student performance;** 


Does the institutional report document 
how the institution informed its students 
about institutional expectations 
regarding student outcomes? 


CRS 23-13-101 (d) **That these 
improvements be achieved efficicndy 
through the effective use of student and 
institutional resources of time, effort 
and money.** 


Does the institutional report include a 
financial statement to document 
institutional accountability costs? 



In addition, the reports must include information about minority student achievement and 
faculty involvement in the accountability process. The following table tabulates each 
institution's success in meeting each criterion. 
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Results 



Most institutions provided data for all criteria; the most difficult criterion to meet appears 
to be "continuing examination of goals". Ten institutions either did not provide dau to 
meet this criterion or misunderstood the question and provided incorrect information. To 
meet this criterion, an institution must compare its assessment results to its goals and 
judge if the goals remain valid. Changes in goals occur because the original goals were 
not sufficientiy demanding, written too narrowly, or may not have expressed educational 
objectives that were originally intended. The CCHE staff, in its annual spring meeting 
with assessment directors, wHl discuss this criterion to clarify expectations. 

Most institutions are meeting the second and third directives of the accountability statute. 
Institutions informed their students of assessment-related expectations and publicly 
announced their assessment results. Some institutions have already provided information 
to local news media informing the public of the accountability results at local colleges. 
(See Appendix 1 and 2 for more detailed information.) 

The cost of accountability decreased from last year's total of $3.1 million. This year, 
estimated institutional accountability costs equaled $2.5 million. This is approximately 
$14.15 per headcount student for the state, approximately $3.50 less per student than last 
year. Two year colleges spent an average of $14.85 per student, universities/colleges 
spent approximately $14.95 per student, and the research universities spent approximately 
$12.27 for each undergraduate student. Both direct and indhrect costs decreased from last 
year and all categories (e.g personnel, travel, etc.) decreased. (See Appendix C for more 
detailed information.) 

During the 1991 Interim Joint Education Committee Hearings, legislators expressed 
concern that institutions were not properly validating their goals with the help of outside 
publics. CCHE will add another criterion requiring all institutions to describe their goal 
validation process, with special attention to external reviewers of accountability goals. 

General Observations 

Data provided by institutions demonstrate that students' knowledge and skills actually do 
increase from entrance to exit. Instigations developed many assessment measures to 
evaluate directiy increases in knowledge due to individual classes, major programs, and 
general education curriculum. All institutions are implementing a plan that is consistent 
with the statute to "be accountable for improvements in knowledge, skills and capacities." 
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The institutions' sophistication in developing and using measures and interpreting data is 
significantly higher than in the previous year* Many more institutions used information 
gathered from assessment measures to make changes in their curricula, teaching styles, 
and requirements for graduation than last year. Faculty at most institutions were involved 
in the process. Some reports relayed information on faculty interest in accountability 
information and how faculty used the data as support for maldng changes. 

One area that most institutions found deficient through their accountability measures is 
advising. Current students and alumni often stated a need for better advising. Many 
institutions responded to this accountability information by providing more information 
and training to faculty to improve their advising skills. 

A second area of concern is the low retention rate at all types of institutions. Changes 
to increase retention include academic alert programs, which identify students with 
academic difficulties early in the semester, and Freshman Seminar classes, which provide 
an introduction to college and the array of services available. 

Assessment of the majors or vocational degrees is progressing very well at the majority 
of institutions. Faculty are closely involved in the processes of de^'doping goals, 
applying measures, interpreting data and changing to solve identified problems. Course 
sequencing, prerequisites, and writing skills are examples of issues identified by 
accountability data and addressed by faculty. 

Review Questions and Institutional Responses 

In addition to reviewing each report to determine if all criteria are met, CCHE also audits 
accountability goals, measures, and results. If CCHE finds unclear or unrealistic 
explanations or measures, the institution is requested to provide more information. The 
following qufrStions and institutional responses are a small sample of all the questions and 
responses received regarding the 1989/90 Accountability report. 

STATE BOARD OF AGRICULTURE 

Colorado State Univer.uty: 

CCHE Question: (To the Political Science faculty, who developed a measure to 
assess this program by asking two questions of their graduating seniors.) Can two 
questions reflect the entire political science curriculum? 
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Response: **First the Department does not seek to test student knowledge of 
the curriculum. Instead, objectives are to sample knowledge in areas central 
to the curriculum and their abilities to analyze politics in an informed and 
systematic basis. 

Further the questions posed are not narrow, and memorization is not tested. 
The Department wishes to know if instruction has given students significant 
relational information and concepts which help them think, analyze, grope 
with difficult questions, make meaningful comparisons, and understand the 
complexities of political life.^ 



University of Southern Colorado: 

CCHE Question: Most of your dissemination procedures appear to be internal; 
more external procedures are necessary. 

Response: "AY 1990-91 was the first year to collect baseline data. During the 
past year our external dissemination consisted of submitting several articles 
to the Pueblo Chieftain, publishing our assessment program and expectations 
in the 1990/92 bulletin and sharing our results to date with area high school 
counselors." 



STATE BOARD OF TRUSTEES 

Metropolitan State College of Denver 

CCHE Question: The processes that many departments developed do not look at 
programs in light of program outcomes, but focus on individual students- 



Response: ^MSCD faculty understand that they are assessing their programs 
in order to improve them. However, faculty choose to assess by examining 
students' performances. To make students take assessment activities seriously, 
faculty believe that the activities and a performance criterion must become 
requirements for graduation. If the activities are required for graduation, 
faculty are faced with the problem of what to do if a student does not meet 
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the criterion. This issue is of grave concern to the faculty, and last year 
faculty may have placed undue emphasis upon this aspect and insufficient 
emphasis on how they would use the information to miproye their programs.** 

Mesa State College: 

CCHE Question: CCHE is concerned about the lack of data reported for majors- 
Response: ^^Disdpline-spednc assessment now uses assessment techniques, 
outcomes, and standards established by each department. The academic goal 
for each area of specialization, however, at this time remains the same as 
outlined in previous reports * that students have ^.•adequate preparation in 
an area of specialization to permit graduate studies ot a professional career.' 
The change in assessment techniques, outcomes, and standards represents an 
expansion of the original assessment program, allowing each area to select 
from a more effective multiplicity of assessment instruments and procedures.** 



UNIVERSITY OF COLORADO SYSTEM 

University of Colorado - Colorado Springs 

CCHE Question: Many departments did not iumish data« 

Response: **UCCS is in complete agreement with CCHE's concerns. We 
realize that some of our departments do not fully understand the 
accountability process. Therefore, we plan to schedule a meeting with all 
deans, department chairs, administration, and staff who are currently 
involved with assessment reporting. During this meeting, we will explain the 
correct format to be used in reporting student outcome assessment;...^ 



University of Colorado - Boulder 

CCHE Question: There seems to be widespread concern about students' critical 
thinking abilities. Are there any plans to address this issue on an institution-wide 
basis? 
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Response: "Some Critical Thinking courses are evaluated now because they 
are also capstone courses used by individual departments in their assessment 
plan. As described further in the discussion of General Education in the 
following section, a more concerted effort will be made to add evaluation of 
Critical Thinking courses m AY 1991-92." 



TRUSTEES OF THE UNIVERSITY OF NORTHERN COLORADO 

CCHE Question: If the Math department considers the California Achievement 
Test (CAT) as too elementary to measure Math skills of sophomores, why is the 
Professional Teacher Education program reporting it as an assessment measure? 

Response: "Furst, the CAT is a state*mandated test for persons seeking 
certification as teachers in the Colorado schools. Second, the CAT is not an 
outcomes measure; rather, it is part of the entrance requirements to the 
Professional Teacher Education Program. Thus, the issues of the level of 
math required on the CAT and level of math skill considered appropriate for 
any UNC graduate are unrelated. Furthermore, the fact that about 1/3 of 
students who apply for entrance to the PTE prc^ram fail the math portion of 
the CAT suggests to them that theur skills may be considerably below what 
will be needed to meet the university's math-skill criterion.'* 



TRUSTEES OF THE COLORADO SCHOOL OF MINES 

CCHE Question: CSM's report appears 1 very administratively based with littie 
faculty input. CCHE is concerned about 'die apparent lack of faculty involvement. 

Response: '^Faculty have been heavily involved since the beginning of the 
process-they formed the bulk of the committee which designed CSM's plan 
and they comprise the current committee which evaluates and reports on 
assessment results. Asjuniors enter the portfolio process for the furst time 
this year (1991-92), the faculty m each department have been asked to devise 
a plan through which the technical competence of their students can be 
measured.** 



STATE BOARD OF COMMUNITY COLLEGES AND OCCUPATIONAL 
EDUCATION 

Otero Junior College 

CCHE Question: Class attendance does not necessarily result in increased 
knowledge and the General Assembly requires accountability plans with measures 
that go beyond reporting grades. OJC needs to develop some additional methods. 

Response: "Table II shows the FaU courses offered and the choice of measure, 
pre-test/post-test or audit, to be used to measure learning during the Fall 
semester." 

Lamar Community College 

CCHE Question: Are faculty incorporating specific employer suggestions into the 
curriculum? 

Response: "The Business Systems Technology faculty have completed m^or 
curriculum revisions that were based on feedback obtained from employers 
within the business community. The Ranch/Farm Management Program will 
be implementing new curriculum components during 1991-92 which directly 
respond to employer suggestions to address the repair and maintenance of 
commonly used ranch and farm equipment." 



LOCAL DISTRICT COLLEGES 

Aims Community College 

CCHE Question: The relationship between some of your objectives (2.3 & 2.4) 
and the assessment measures is unclear. (Objective 2.3 - Think critically about a 
variety of human issues at success or minimum competency levels established by 
the faculty in the division and course work. Objective 2.4 - Weigh and respect 
different systems of human values, both contemporary and historical, making 
appropriate connections between past, present and future at success or minimum 
competency levels established by the faculty in the division and course work.) 
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Response: meetings with the Behavioral and Social Sciences Division 
regarding this issue, the scope of performance expectation 2.3 was determined 
to be somewhat too general in relation to the specific curricular content 
assessed. The performance expectation have been revised to : "Think 
critically about psychological, sociological, and geographical issues at success 
or minimum competent levels established by the faculty in the division and 
course work.'' Also the faculty of the Behavioral and Social Sciences Division 
are now reviewing both performance expectations 2.3 and 2.4 relative to the 
acquisition of content versus demonstration of behavioral change." 



STATEWTOE POLICY ISSUES 



The institutional accountability reports provide two views: insututional information 
and a view of issues statewide. The following issues appear to be ones that may 
have statewide implications. 

1) Effectiveness of Basic Skills - Three institutions reported information on 
students assessed as needing basic skills. One group of students took the advice 
to take remedial courses; the other group did not tate the advice. The institution 
then compared the success of each group in college level courses. Although those 
that took the remedial courses had higher success rates, the differential is not large 
in most cases. I1ie following data provides the support for this issue. 



Students Assessed As Needing Basic Skills 
Percent of Students Successfal in College Level Courses 



OJC CCA TSJC 

Remedial No Remedial No Remedial No 

Course Course Course Coujse Course Course 

English 71% 67% 78% 67% 91% 88% 

Math 67% 27% 89% 55% 96% 100% 



Took Remedial N = 384, No remedial N = 197 



Recommendation: Only the three schools identified above reported this kind of 
information. This group is too small to provide definitive evidence of the 
effectiveness of basic skills preparation, but it does provide enough evidence to 
suggest further exploration of this issue. Ideally each institution should be 
addressing this issue, but the demands of this kind of research project may be 
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beyond the resources of some community colleges. The community college 
governing board does have the resources and this is an issue that should be 
addressed by them. Depending on the results on the governing board study, 
CCHE may further address this issue. Responsible entity: SBCCOE and CCHE. 

2) Assessment of Teacher Education Programs - Institutions have not developed 
meaningful assessment techniques for teacher education programs. Most programs 
rely on some or all of the following: California Achievement Tests (an entry test 
required by the Colorado Department of Education), the CDE 1st and 3rd year 
survey of new teachers, student teaching grades, and the CDE survey of school 
districts. None of these assessments distinguish between programs, nor do they 
provide much detail to individual programs since most percentages hover around 
90%-9S % level for all questions at all institutions. Most institutions therefore find 
the need to change very littie in their programs. Because of the inadequacy of 
these assessment measures, it is difficult to tell if programs should be changed. 
Basically, these assessments support the status quo. 

Recommendation: Teacher education assessment should reflect current thinking 
about assessment. Current assessment results are not sensitive enough to 
distinguish between programs or apparentiy give enough information to institutions 
to suggest substantive changes in programs. Assessment techniques are the 
responsibility of individual institutions. Linkages in Networking Colleges and 
Schools (LINCS), a joint committee established by CDE and CCHE, and recentiy 
adopted as a task force of the Colorado Achievement Commission, could be an 
excellent resource for education deans to share views of statewide needs at both 
the K-12 and higher education level. Responsible entity: Education Deans and, as 
a resource, LINCS. 

3) Minority Retention/Gradua tion - The minority graduation and retention rates 
are, at almost all institutions, signiticantiy lower than for non-minority students. 
CCHE studied this issue in the last year and produced a report detailing retention 
and graduation rates for all public higher education students in Colorado. The 
study found that from fall 1986 to fall 1990, minorities persisted or graduated at 
lower rates than non-minorities at both community colleges and four year 
institutions. This information will be updated annually. 
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Persistence/Compietion Rates for Colorado Public Higher Education* 

Community Colleges* Four Year Institutions 



Blacks 


26% ■ 


45% 


Asians 


40% " 


68% 


Native Americans 


36% 


47% 


Hispanics 


36% 


55% 


Whites 


47% 


65% 


Total 


44% 


64% 




N=3,912 


N=9,029 



* See Persistence/Completion Rates for Colorado Public Higher Education - 
CCHE November, 1992 Agenda Item IV, B for methodology. 

The Commission has a statutory directive to develop statewide affirmative action 
plans and has instituted an affirmative action program that sets minority graduation 
goals for institutions. The graduation rate for minorities will be monitored each 
year and compared to previously set graduation goals. This is an area where 
institutions know that they must make progress to serve fully the entire population 
of Colorado. 

Recommendation: CCHE should continue to monitor the retention and graduation 
rates of minorities and publicly report those results. Responsible Entity: CCHE 

CONCLUSION 

This report documents CCHE's statutory responsibility to monitor institutional 
accountability efforts and to report on those efforts to the legislature. All 
institutions fulfill the mandate to measure the increase in knowledge, skills, and 
capacities that occur between entrance and graduation. They also are using the 
information to mate changes in their classes, teaching methods, and programs. 
The changes arc too numerous to document in this report, but can be obtained 
firom the fiill reports available from CCHE or the institutions. 



' In comparing two year and four year persistence/completion rates, it is important 
to remember that two year students have a wide variety of objectives. Many students 
with certain goals (e.g. job skills upgrade, personal improvement, courses to obtain a job) 
do not intend or want a degree, but fulfill their objectives by taking courses only. 
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The institutional reports reflect some statewide issues that need further 
clarification. Basic skills, minority retention/graduation, and teacher education 
assessment are issues that need additional study from a statewide policy 
perspective. 
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ESTIMATED COSTS OF ACCOUNTABILITY 



Costs - statewide 

$2,406,789 Total 

$14.15 Per Student 



Direct Costs 

Test/ Surveys 
Equipment/ Computer 
Indirect 
TOTAL 



$1,487,978 
122,822 
5,600 
844.389 

$2,406,789 



Costs per student (1990/91 headcount) 
Rttsaarch University Sector Community Colleges 



CSM - 


$15.71 


OJC - 


$ 9.94 


UCB - 


9.93 


RRCC - 


6.82 


CSU ~ 


12.83 


FRCC - 


15.19 


UCHSC 


35.28 


CCA - 


8.27 






ACC - 


19.58 




avg. $12.27 


PPCC - 


17.38 






LCC - 


17.13 






MCC - 


23.81 


Universities & Colleges 


CCD - 


23.84 






PCC - 


36.60 


UCD - 


$ 7.91 


TSJC - 


38.60 


MESA - 


13.24 


AIMS - 


$ 3.93 


UNC - 


15.70 


CMC - 


5.39 


use - 


21.54 


NJC - 


33.19 


uccs - 


11.68 


CNCC - 


15.78 


wsc - 


4.76 






MSC ~ 


14.19 




avg. $ 


ASC - 


36.26 




FLC - 


28.03 







$14.85 



avg. 



$14.95 
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